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NICOS, Nightly COntrol System, is a flexible tool for coordination of software development in large-scale 
projects. It manages the multi-platform nightly builds based on the recent versions of software packages, tries 
to compensate for technical failures, tests the newly built software, identifies possible problems, and makes 
results immediately available to developers spread over different institutions and countries. The NICOS nightly 
build services ensure that new software submissions are consistent and provide expected results. The NICOS 
tool was developed to coordinate the efforts of more than 100 developers from 34 countries for the ATLAS 
project at CERN and can be easily adapted for other large software projects. 



1. Introduction 

The software projects for High Energy Physics ex- 
periments are international with often several hun- 
dreds programmers from institutions world-wide in- 
volved. The coordination of distributed software de- 
velopment is an important factor of success. The au- 
tomated nightly builds become a major component 
in collaborative software organization and manage- 
ment. In multi-person, multi-platform environment 
they provide a fast feedback to developers on new code 
submissions and facilitate collective work on different 
or same parts of software. 

NICOS Q was originally created for the ATLAS ex- 
periment 0] and evolved in a versatile nightly builds 
system. It operates on UNIX-like platforms and 
works with known release management tools, such 
as SCRAM and CMT ||. The NICOS nightly 
builds become a media in which advanced program- 
mers perform code development and make this devel- 
opment more effective and high-quality. Currently the 
NICOS tool is also used in the LHC Computing Grid 
Project [5j- 



2. Goals and Design Principles 

The task of NICOS is to provide a nightly build 
system that 

• works with known software release tools, 

• provides options for version management and 
number of releases in a cycle, 

• performs testing of the newly built software, 

• informs programmers about results with dy- 
namic web pages and personal e-mails. 

The design strategy is based on a goal to make the 
system flexible, portable, stable, and easy to use. 



Flexibility is attained by organizing NICOS in a 
modular way so that each module is responsible for 
a certain step and independently described in the 
NICOS configuration file. The modular structure al- 
lows to create the nightly builds framework that allows 
to plug in build, test, validation and other external 
tools. For the task of software management NICOS 
relies on external release tools 0, Q . 

NICOS is based on PERL scripts that makes possi- 
ble porting to UNIX and Windows systems. NICOS 
creates web pages that can be viewed with major web 
browsers, such as Netscape, Internet Explorer, and 
Opera. 

The NICOS system include the peer process that 
controls the execution of modules. It detects prob- 
lems and makes an attempt to rerun failed modules. 
This feature allows to compensate for temporary tech- 
nical problems and achieve maximum stability of the 
nightly builds. 

NICOS is easy to use for both administrators 
and software developers. The NICOS project con- 
figuration is stored in one unique text file named 
nicos_cache. The versions of packages for the nightly 
builds can be specified in the NICOS database files or 
supplied by an external tool. A fast feedback to de- 
velopers is one of the most important goals of nightly 
builds. NICOS automatically posts the information 
about the progress of nightly builds, identifies prob- 
lems, and creates the web pages with build results. 
The authors of failed packages can receive automatic 
e-mail notifications if desired. 



3. NICOS structure 

NICOS organization is shown in Figure ^ The 
main process, NICOS Controller, can be run at sched- 
uled times by system tools, such as cron. It runs the 
NICOS job modules and directs the status informa- 
tion to the NICOS project web page. If desired, the 
previous nightly release is preserved and new release 
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Figure 1: Nicos modular organization. 



is built in a separate area. The number of releases 
in a cycle is determined by the NICOS administra- 
tor. NICOS supports multi-platform builds (starting 
version 0.2) sharing code sources. The main build pro- 
cess checks out code and triggers the start of builds 
on other platforms. 

In case of failure the NICOS Controller tries to 
make a restart from the point of failure. If the NICOS 
job runs overtime (typically more than 24 hours) it is 
automatically destroyed. 

There are eight major modules in the NICOS job. 
They are able to serve as the interfaces to external 
tools, such as CVS, test, and release management 
tools. 

• NICOS configuration. At this step the project 
parameters are read from the nicos_cache file. 

• Release tool setup. The release tool (such as 
SCRAM, CMT) is setup with commands speci- 
fied in nicos_cach.e. 

• Code checkout from the CVS repository speci- 
fied in nicos_cach.e. The packages tags are re- 
trieved from the NICOS database file or supplied 
by an external tool. 



• Project setup. At this step the project setup 
commands can be specified. 

• Project build. The release tool should insert 
separators between the packages builds outputs 
with the packages names. The unique string 
from the separators should be specified for this 
step. After the build NICOS cuts out and writes 
the logfiles with the build outputs for individual 
packages to the Log area of the project. 

• Unit tests and Integrated tests. The test tools, 
such as OVAL @ or CppUnit are plugged in, 
as desired. The outputs are forwarded to the 
TestLog area of the project. 

• Error Analysis. In the build output NICOS 
searches for critical patterns indicating make 
problems, such as 'No rule to make target' 
or 'Symbol referencing errors'. For tests, 
success is determined by the value returned by 
test tools. 

• Creation of the web pages with the summary of 
build results and e-mail notification about prob- 
lems. 

The progress and results of NICOS builds are re- 
flected on the dynamically created web pages. Its 
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Figure 2: Nicos web pages. Secured pages are marked with lock sign. 



structure is shown in Figure The NICOS project 
web page shows the major parameters, such as project 
area, compiler options, and list of releases available 
for supported platforms. For building releases, the 
currently performed step is displayed. The NICOS 
project web page contains links to the pages with de- 
tailed results for individual releases and pages showing 
the NICOS database and configuration parameters. 

Optionally the collection of PHP scripts, "NICOS 
Webmaster" , can be used for handling the admin- 
istration of NICOS over the WWW (available in 
0.2 version of NICOS). This tool can modify the 
project configuration (' 'Project Configurator'') 
and NICOS database of package versions (' 'Version 
Administrator ' ' ). In addition it is possible to sched- 
ule or stop builds from the Job Controller page. 



4. Configuration Management 

The configuration parameters for the NICOS job 
is collected in the XML-like nicos_cache files. The 
steps of building process is associated with its markup 
tag. Every tag consist of a tag name, sometimes fol- 
lowed by an optional list of tag attributes that are 
placed between brackets ( < and > ). The markup 



tag is followed by the commands, including environ- 
ment definitions, needed to be executed at the step. 
The example of a markup tag below is for the step of 
build. The attribute defines the directory name (rel- 
ative to the project area) where the build should be 
performed. The tag is followed by command scram b 
that builds a SCRAM based project. 



<project build dir=src> 
scram b 



NICOS checks out versions of packages as speci- 
fied in the NICOS database file. Each line of this 
text file contains the name of package, tag of package, 
and e-mail addresses of the developers separated by 
spaces. The exact tag of package or option for tag 
calculation can be indicated. The options include the 
selection of recent submission, recent tagged submis- 
sion, recent version in the "official" format: <package 
name>- [0-9] [0-9] - [0-9] [0-9] - [0-9] [0-9] . The 
NICOS database file can be indicated in nicos_cache 
or dynamically created by a plug-in script. The latter 
option is useful for large collaborations where special 
version management tools are used 0. 
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5. Status and Plans 

The 0.1 version of NICOS is available since March 
2003 and provides basic functionalities. New 0.2 ver- 
sion (scheduled for September 2003) will contain im- 
portant improvements (e.g. multi-platform support) 
and additions such as the PHP script collection for 
administration over the WWW. The long term plans 
center around improving simplicity of use and porta- 
bility (including the support of NICOS for Windows 
OS). 

6. Conclusions 

NICOS (Nightly COntrol System) is currently suc- 
cessfully used by several software projects at CERN 
(e.g. ATLAS, LCG). The NICOS features such as 
the instantaneous error analysis, dynamic informa- 
tive web pages, and e-mail notifications about prob- 
lems are proved to be useful for collaborative software 
organization and management. As experience gains, 
NICOS undergoes continuous improvements and evo- 
lutions. 
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